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[57] ABSTRACT 

Management packets are defined that are modified in the 
payload by each node along a virtual connection and are 
used to measure both end-to-end QoS and specific individual 
intermediate node performance parameters. Management 
packets are implemented by defining entirely new packets or 
by modifying ATM OAM cells. Switches or routers for use 
as intermediate nodes are defined that modify the payload of 
the management packet and locally measure packet delay 
and packet loss. An intermediate node measures and records 
the difference between the arrival and departure times of 
management packets at that switch utilizing delay-stamp 
fields within the management packets and either the switch 
internal routing header or timestamp fields within the 
packet At the endpoint of the virtual connection, delay- 
stamp fields in the management packet indicate individual 
node delays and the cumulative delay. An intermediate node 
counts the number of packets it discards and records these 
values in the payload of the management packet individually 
and cumulatively. 

19 Claims, 7 Drawing Sheets 



Time -of- Day 
Stamp 



\ 

40 



SWITCH 



Delay 
Stamp 



\ 

46 



36 



32 



\ 

34 



»> 



Input 

Header 

Processor 



38 



Switch 
Fabric 



42 



Output 
Header 
Processor 



44 



31 



07/30/2004, EAST Version: 1.4.1 



U.S. Patent 



Aug. 11, 1998 



Sheet 1 of 7 



5,793 



o 

Q 
Hi 



o 

CD 
Q_ 

CO 
i 

c 
o 

o 
c 



(0 
CD 



Q. 

E 
iS 

CO 

E 



CO 

CO ^ 

= o 
U 



Q CO 



c 

-2 „ 



s = s 



to 

<D 



8 



00 



0) 



CVJ 



■ MB 

UL 



CM 



07/30/2004, 



EAST Version: 1.4.1 



U.S. Patent 



Aug. 11, 1998 Sheet 2 of 7 



5,793 




07/30/2004, EAST Version: 1.4.1 



U.S. Patent 



Aug. 11, 1998 



Sheet 3 of 7 



5,793 




07/30/2004, EAST Version: 1.4.1 



U.S. Patent Aug. 11, ms sheet 4 of 7 5,793 




07/30/2004, EAST Version: 1.4.1 



U.S. Patent Aug. ll, 1998 Sheet 5 of 7 



5,793 




07/30/2004, EAST Version: 1.4.1 



U.S. Patent 



Aug. 11, 1998 



Sheet 6 of 7 



5,793,976 



oo 




<0 

ri> 



SO 



<N 



07/30/2004, EAST Version: 1.4.1 



U.S. Patent 



Aug. 11, 1998 



Sheet 7 of 7 



5,793,976 



oo 



o 

t? g 

O 3C o- 



SO 





--> 


Contr 
Proces 


< -- 



CO 



0> 

■s 



o 

(A 

v> 
o 



o 



CO 

in 



ri> 
iZ 



o 



07/30/2004, EAST Version: 1.4.1 



5.793,976 

1 2 

METHOD AND APPARATUS FOR taining both the QoS per virtual connection and the efficient 

PERFORMANCE MONITORING IN utilization of the network. Performance management itself 

ELECTRONIC COMMUNICATIONS depends largely on the data obtained through performance 

NETWORKS monitoring, which is conducted through measurement of the 

5 QoS per VPC/VCC Because of the use of statistical mul- 

FIELD OF THE INVENTION tiplexing in ATM networks, the QoS of a virtual connection 

The present invention relates to electronic communis- ^ ^Particular be adversely affected by *e random flow 

tions networks and. in particular, to performance monitoring °{ tra f Bc on the n ?™*: IS by a number 

of packet-switched data networks. of «te faaoB. including network congestion and trans- 

to mission faults, as well as intermittent error conditions that 

BACKGROUND may cause a gradual deterioration in QoS. Network provid- 

_ _ „ . , . „ „. . . „ , ers need to be able to detect and diagnose troubles quickly 
The Broadband Integrated Services Digital Network mor<SelV) minimi2e the effect of toose ttoublcs on network 
(B-ISDN) can accommodate a variety of packet -oriented usefs 

protocols, including asynchronous transfer mode (ATM), in _ ' , . t ^ 
which standardized 53 -byte data packets are utilized |mj-T 15 Performance monitoring can be done either through out- 

Rec. 1.121, Broadband Aspects of ISDN. Melbourne. of-service testing or m-semce QoS monitoring. Oui-of- 
November 19881. ATM and some other packet-oriented scrvicc testing involves P^ornmnce measurements made 

protocols are connection^>riented. utilizing what are known under controlled traffic conditions. TTus type of testing has 

in ATM as virtual path connections (VPC) and virtual ^ a number of drawbacks, not the least of which is that service 

channel connections (VCC) that stretch between two end- 20 0D network must be interrupted. In addiuon, the results 

points across transmission links connected by intermediate received are only as accurate as the model used to create die 

nodes. These intermediate nodes are devices that forward controUed traffic conditions. It is entirely possible that the 

and direct the packets to their intended destination endpoint contioUed conditions tested may not correctly model some 

i.e. routers, or switches. Packet-switched data networks are „ of toc conditions actually seen by the network during 

capable of supporting a simultaneous mix of voice, video. 25 * addltl0a / tms does no ' ^ r fP ,d 

anddata services over either constant bit rate (CBR) or dctccllon of «**q««it degradauon of the network, 

variable bit rate (VBR) connections. In-service QoS monitoring involves continuous perfar- 

For each virtual connection, an ATM network can guar- measurements on actual user traffic. No service 
antee to the customer a particular level of performance, ^ interruption occurs and network degradation is instanta- 
known as a quality of service (QoS). that is negotiated neous, y detected Continuous performance monitoring is 
between the user and the network during connection set-up. particularly important in high-speed networks where service 
Once agreed upon, the QoS can only be changed through deterioration may quickly affect a large amount of user data, 
renegotiation between the two parties. In general, the delay In-service QoS measurements are also useful to supplement 
experienced by a data packet as it travels along a network 33 traffic comro1 algorithms based on assumed traffic models, 
connection is an important measure of performance for a such « used in out-of-service testing, that may not 
packet-switched network. QoS in ATM is specified mainly accurately characterize the actual traffic, 
in terms of acceptable cell delay and acceptable cell loss rate Current ATM standards provide for the measurement of 
[ATM Forum. ATM User-Network Interface Specification cell delays and cell loss using Operations and Maintenance 
Version 3.0. Sep. 10. 1992]. ATM supports mumple QoS 40 (OAM) cells [ITU-T Rec. 1.610, B-ISDN Operations and 
classes geared to differing application requirements for Maintenance ftinciples and Functions. Geneva. June 1992; 
delay and loss performance. A VCC or VPC is therefore Bellcore, Generic Requirements for Operations of Broad- 
characterized by its requested QoS and throughput param- band Switching Systems, TA-NWT-001248, issue 2 Oct. 
eters. A VPC may carry statistically multiplexed VCCs of 1993]. OAM cells are specialized versions of standard 
differing QoS classes, but the overall QoS of the VPC must 45 53-byte ATM cells, having a normal 5 -byte header with a 
satisfy the most demanding QoS of the carried VCCs. 48-byte information field, but carrying management infer- 
Specified QoS parameter values may be either explicitly mation within the function-specific information fields 
requested by the user or implicitly associated with a par- instead of user data. ATM user cells themselves cannot be 
ticular service. used for QoS monitoring because the payload is full of user 

An important concern is management of traffic to ensure w **** **** no neadcr ficids arc available to receive the 
that users receive their guaranteed QoS. The assurance of a information mat is needed by the performance monitoring 
particular QoS is a fundamental part of the contractual system for calculation and storage of cell loss and cell delay, 
agreement between the network and the users because OAM cells are routed by the ATM network in the same 
certain applications critically depend on receiving a particu- manner as user cells, but are distinguished from user cells by 
lar QoS from the network. Voice transmissions, for example, 55 the network by the assignment of special values to the VCI 
are particularly sensitive to end-to-end delay. For this and/or PT fields in the 5-byte cell header, utilizing VCI=3. 
reason, public telephone networks have a m^Ti'mnm allow- VQ=4, FT=4 or PT=5. OAM cells for performance man- 
able delay specification of 600 ms I Chen, Thomas M.. agement are further identified in the information field by a 
Walrand. Jean, and Messexschmitt, David G., "Dynamic 4-hit OAM Cell iype=0010. A 4-bit OAM Function Type 
Priority Protocols for Packet Voice.** IEEE Journal on $o field in the information field further indicates the specific 
Selected Areas in Communications. Vol. 7, No. 5, June 1989, performance management function performed by the OAM 
pp. 632-643]. with any delay greater than 15 ms generally cell, such as forward monitoring (0000). backward reporting 
requiring the use of echo cancellation IMcDysan. David E (0001). or both (0010). A 10-bit EDC field contains an error 
and Spohn. Darren L.. ATM: Theory and Application. detection code that is computed over the information field of 
McGraw-Hill. 1995. p. 355|. 55 the cell. 

Performance management, as a function of network The forward monitoring procedure consists of inserting 

management, has the simultaneous responsibilities of main- OAM performance management cells into the ceil stream 
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between blocks of user cells"* one endpoint of the virtual altered greatly between the two halves of the OAM cell's 

reporting Solves putting observed trip. fiirthermore. this >memod is also notable to measure the 

oerformance data into an OAM cell at a virtual connection delay occurring at individual intermediate nodes. ^ 

endpoint and then sending the OAM ceil backwards in the All of these disadvantages to current network pcrfor- 

cellstream direction from which it came, in order to report 5 mance monitoring methods arise generally from the tact that 

the oerformance of that particular virtual connection to the the intermediate nodes in the virtual connection relay the 

oriXating endpoint. OAM cells but do not actually modify their payloads. 

TomeaLTll .os, the OAM cell contains the number AUhough ATM switches^ haverhe 

. • ^ .a: u\~Mr nv.tfti n<^r oil and manipulate the header of an AIM ecu. it is noi normal 

The major disadvantage of the current network perfor- e . ^ rf me yimal ^^0,,. The performance moni- 

mance monitoring method for measuring cell loss is tnatit ^ bm of me netwolk h severely limited by this 

cannot be assured of actually determining the exact number ^ * f g ^ ity £ m(etmedi!lU . node$ , 0 add information into 

of cells lost. This is because cells can also ^^uerted m 0 f a passing data packet What has been needed 

into the cell stream. Therefore, for example. If the OAM cell . $ ' ^ aQd utmK ftc illformatk)ll ao be 

indicates that N user cells were sent and then N-l cells are ^ performance monitoring system by the 

actually received, it is not possible to be sure whether 1 ceU ^ tennediate D00e f 

was lost, or whether 2 cells were lost and 1 misinserted. and ,DBnut,1M ' 

soon.ltisalsonotpossftle.usingthfaraemc4.todcterniine ?j Objects of the Invention 
from where along the virtual connection, or from where Accordingly, a primary object of the present invention is 
within a cell block, the cell loss actually occurred. t0 provide accurate, instantaneous monitoring of the perfor- 
in order to measure cell delay, the OAM performance mance of a packet-switched network connection, 
management cell contains an optional 4-byte Timestamp ^ particular, an object of the present invention is to 
field that is used to indicate the time that the OAM cell was M mon jtor and report the delay experienced at each interme- 
inserted into the user stream. The timestamp is then com- fl^e node of a network connection by a packet traveling 
pared with the arrival time of the cell at the virtual connec- fte connection, as well as to calculate accurate end- 
tion endpoint and the difference is inferred to be the end- uvend delay along the connection, 
to-end delay experienced by a cell along the virtual A particular object of this invention is to accu- 
connection. However, this type of timestamp is meaningful 3J rJte)v monitor and report the number of packets lost or 
at the destination only if the clocks at the two endpoints of discarded by each intermediate node in a network 
the virtual connection are synchronized to the same time- connection, as well as to accurately calculate the number of 
of-day. something which is generally not practical packets lost end-to-end. 

In addition, the existing method of delay measurement SUMMARY 

does not take into account the fact that end-tc-end delay « 

legally consists of two distinct types of delay: fixed propa- In one aspect of the invention, a new das .of packet* 

Nation delay which is that delay introduced into the con- defined which has a payload that .s modified by each node 

Xn * a predictable level by the transmission links along .virtual ™^J"?*^ ZnVa 

'™1 bctweeTthe network nodes, and random delay. accurately measure and report the end-to-end QoS along a 

Random delay is that delay which is introduced by each of « virtual connection, as well as to measure and report specific 

Se^« nSs Ae^selves. and can vary widely with performance parameters at individual -^^nodes.In 

nework traffic levels and switch buffer capacity. In general. an ATM eintadiment management fWs are implemented 

u^Ttype of delay is primarily due to queuing delays, which by defining an entirely new type of ceU orby defining 

STtacW wiHncreLd traffic load. The existing additional fields within existing OAM ^J^™"** 

method of performance monitoring using OAM cells is not » each mtermediate node modifies Aepayloadof management 

abSto £ackttie delay that is actually eVrienced at each packets and locally measures delay and packet to* .A 

ZL nong the virtual connection due to such factors. function type field specifies fce particular performance 

B t alternative way to measure cell delay is to me.ure ^^^^XST^XS 

TSSt^-^vdSS^L^L between field is incremented by the local delay measured at each 

□^es^The^ayead to^nd delay is then estimated as switch. When the management packet reaches *e endpomt 

hXme VLTtiip dT y . However, while the loopback cell of the virtual connection, me^delay-starr^ field mdicate the 

Sd 2 meas P ure the round-trip delays without need for cumulative amount of tune that the packet has been delayed 
dock synchronization, it gives only an estimate of the 65 by the nodes of the network. 

oneway ceU delay. This estimate may in actuality be very In another aspect of the invention a node-by-node delay 

inaccurate if the network congestion or characteristics have measurement management packet allows computation and 
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relay of instantaneous delay information by utilizing mul- Nod- associated management packets are extracted from 

tiple delay-stamp fields to record each delay encountered on the data stream and diverted to a special processor, therefore 

a node-by-node basis. At the destination endpoint. die end- not retaining their positions relative to the user data packets 

to-end delay is computed as the sum of die recorded delay- j n the same virtual connection. Non-associated management 

stamp fields. Alternatively, the switch may record in one 5 packets are used in one aspect of the invention by a control 

timestamp field the time that the packet arrives, and then processor to report cumulative performance or usage mea- 

record the tune the packet leaves into another times tamp surements. The input header processor creates an internal 

field, with the calculation of node-by-node delay made when routi header for no . B _ assoc i ate d management packets that 

the packet reaches the endpoint of the virtual connection. iUeas , hc mmi fabric t0 route me p^kcls , 0 At . co,,,^ 

According to a further aspect of the invention, interne- ,„ processor for modification, 

diatc switches car. count the number of packets each dis- Uot-assocUtcd management packets are used in another 

cards. When a node receives an end-tc~end loss accumulat- rf ^ Nation t0 local measurement of 

mg managementpacket. it .ncreir^ntsj, packet loss count ^ ^ ^ ^ , osscs „ re ^ adll m£ no „. 

field by the number of packets discarded by the node within ^ ^ ^ temporarily witi a marker 

that virtual conne^on since the previous loss management 15 (ha( ^ of fcTnVn-associated manage- 

packet arrived. When the end-tc-end loss accumulating ^ [a ^ J^ saesm ^ me ^ 

management packet arnves at the vmuaj ooDnecuon reew £ , the dela or packet loss count information in the 

C ^ i? 1 ^, T^* payload of the rLiagWnt packet. The marker packet is 

«^ S f d4 l pa , CketS m preCetUng bl0Ck ° f us «:P acke,s - STen replaced iofcedata strWwith the modified non- 

Node-oy-node loss measurement management packets con- M assodate dmana jsement packet Local measurement of delay 

tain tmiltiple packet loss count fields for recording fte ft^t^auh implemented in an alternate embodiment 

5 UD ^ ^ ** ? . 2*. of meinvention by witting all data packets through a control 

departure of the previous loss management * processor, recording delay and packet loss count informaUon 

vutual connection. The total packet loss is then calculated at ^n^^p^cu. and performing 

the endpotnt of the vutual connection. „ ^^J^ t0 user ^ u ^ 

Associated management packets experience the same 
delay and packet loss rate as the user data packets in a BRIEF DESCRIPTION OF THE DRAWINGS 
particular virtual connection, implemented in one embodi- 
ment of the invention by having the management packets mG - 1 is a Wock diagram of an embodiment of the 
follow the same path as the user packets through the switch, 30 management packet of the present invention for an ATM 
The payloads of associated management packets are pro- implementation; 

cessed without the packet being removed from the data FIG. 2 is a block diagram illustrating one embodiment of 
packet stream. In one embodiment, this is implemented by the method of the present Invention, utilizing end-to-end 
using the switch internal routing header. When the packets delay accumulating management cells; 
enter the switch input header processor, it uses the network 35 FIG. 3 is a block diagram illustrating another embodiment 
routing instructions for each packet to create an internal of the method of the present invention, utilizing node- by- 
routing header that for a delay management packet includes node delay measurement raanagement cells; 
the time of entry into the switch. In an alternate embodiment, FIG 4 is a block diagram illustrating an embodiment of 
the time a delay management packet arrives at the switch is thc ^ ^ m invcIltioiK utilizing end-to-ead 
recorded directly into a timestamp field in the packet while 40 ioss accumulating management cells; 
the previous user data packet is being sent to the switch __ _ . ~ ... ^_ . ■ . 
fabr f ^ B FIG. 5 is a Wock diagram illustrating another embodiment 

_ ' - . . . _ „ ^ . , . , . . of the method of the present invention utilizing node-by- 

In the former embodiment, all the data packets are routed . m . o ™„.. 11 ™^ f ,; mflnanomAnt ^ Ile . 

through the switch fabric and directed towards their intended ^ loss measurement management ceUs, 

output ports through an output header processor, which 45 ? G 6 15 a block diapam^ustratmg the flow ofassoa- 

removes the internal routing header from the packets, com- atcd management packets through a switch with local 

pares the time-of-day stamp in the internal routing header of observability of packet delay according to one aspect of the 

delay management packets to the current time-of-day to present invention; and 

calculate the delay incurred by each packet in the switch, FIG, 7 is a block diagram illustrating the flow of non- 
and records the difference in the delay-stamp field of the 50 associated management packets through a switch according 
management data packet This occurs while the previous to another aspect of the present invention, 
user data packet in the data stream is being sent to the output 
port so that the management data packet retains its place in 
the data stream A similar procedure is followed if the 
timestamp field is used, with the output header processor 55 The limitations of prior art network connection perfor- 
reading the time of entry from that field. mancc monitoring methods are eliminated by defining a 
In another aspect of the invention, packet loss counts are special class of packet, a "management packet." having a 
recorded in loss management packets. When a loss manage- payload that can be modified by each node along a virtual 
ment packet is routed through the switch output header connection. This method can be applied to any packet- 
processor, the processor records into the packet loss count 60 switching network. In the preferred ATM embodiment these 
field the number of packets discarded by the switch within "management cells' 1 can optionally be implemented as 
that particular virtual connection since the last loss manage- extensions of current OAM cells. Like OAM cells, manage- 
ment packet was processed. The packet loss count is ment cells carry measurement data along the virtual con- 
recorded while the previous user data packet in the data nection in the cell information field. The major difference 
stream is being sent to the output port, allowing the loss 65 between management cells and prior art OAM cells is that 
management data packet to retain its place in the data the information field of a management cell is modified by all 
stream the network nodes along a virtual connection, not ju st by the 
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virtual connection endpoints-'This new capability makes it known as "local observably" of <^ » e " u " 

rabble for the performance monitoring system to track the most switches have at least some buffering capaWi ty. Jbt 

rather than just the end-to-end performance of the virtual vmjbfc de T ay wUhin a switch tends 
connection as a whole. to increase a s network traffic increases. Existing packet 
Management packets are used to accurately measure the sw i tcncs are not currently implemented so as to have the 
end-to-end QoS experienced by the user packets along any ahai(y to ^ fe^y experienced by a packet within 
virtual connection. In addition, specific performance pararo- ^ swiJch itsclf ^ management packet method utilizes 
eters measured at individual intermediate nodes are made measurements of the individual packet delay experienced 
available to the perforniance inomtoring systeia Tms allows 1 ^ input output of each of the intermediate 
in-service monitoring to be used to accurately identify the sw jtches. Existing switches used with performance moni- 
QoS actually seen by users and to detect exact causes of tQfillg 0AM ha VC the ability to timestamp OAM cells 
performance deterioration before users are seriously wncp catcr a ^ stream. In the management packet 
affected. The method of the present invention therefore method, this activity can be replicated at each intermediate 
offers vastly improved in-service monitoring of both end- 13 Qodc ff mc management packet is an end-to-end delay 
to-end to-end packet delay and packet loss rate. accumulating management cell, the intermediate node takes 
The method requires that the router or switch at each node ^ Delay-stamp field 10, which was set to zero at the 
have specific, umque capabilities^^ originating endpoint of the virtual connection, and incre- 
of management packets and local measurement of packet nients it by the local delay measured at that switch, as 
delay and packet loss. Management packets are used to illustrated in FIG. 2 for an ATM embodiment, 
accurately measure end-to-end packet delay, without the ^ pj G 2 , the Delay-stamp field 10 initially contains a 
need for time-of-day clock synchronization between net- delay TO from previous intermediate nodes. At nodel 16. the 
work connection enefcoints. by measuring and reporting the me ^ of pi G j enters the node 16 is recorded in the 
packet delay at each node along a virtual connection. ^ Timestamp field 9. This field is men compared to the time 
Likewise, management packets allow accurate measurement ^ ^ ie aves me DO de and the Delay-stamp field 10 is 
and reporting of both end-to-end packet loss and packet loss incremented by the difference. TL At node2 18. a new time 
at each node along a virtual connection. i n ] s recorded in the Timestamp field 9. corresponding to the 
The ATM management cells of the preferred embodiment time the cell has entered the node 18. This field is again 
may be implemented either by defining entirely new cells, or M compared to the time the cell exits the node 18 and the 
by defining additional fields within existing OAM cells, difference, T2 is used to increment the value in the Delay- 
since existing OAM cells used for performance management stamp field 10. A similar procedure is followed at node3 20, 
have 33 bytes in the information field that are unused. An and so oo. until the endpoint of the virtual connection is 
embodiment of a management cell for an ATM application reached. Alternatively, the time a packet enters each node 
is shown in FIG. 1. A 53-byte ATM management cell has a J5 can be incorporated into a field in the internal routing header 
standard 5-byte ATM cell header 2. Within this cell header used by that node to direct the packet to the proper output, 
2 the cell is identified to the network as an OAM cell by making use of a timestamp field within the packet unnec- 
specific codes in the VQ and/or FT fields (VO=3, VCI=4. essary. The switxfa then examines the time of entry recorded 
ET=4 or PT=5). In an alternate embodiment, entirely new i D the routing header to calculate the delay that is written or 
VCI and FT codes are defined that uniquely identify the cell ^ incremented into the Delaystamp field 
to the network as an ATM management cell. Following either procedure, when the management cell of 
If existing OAM VCI and FT codes are used in the cell FIG. 2 finally reaches the endpoint of the virtual connection, 
header 2 the management cell is instead uniquely identified the Delay-stamp field 10 indicates the cumulative amount of 
as a management ceU to fceiietwo*^ time that the cell has been variably delayed by the nodes of 
4-bit OAM Cell Type field 4. A 4-bit Function Type field 6 45 the network. If desired, this delay can then be added to the 
further indicates the specific performance nwnitoring rune- value of fixed propagation delay caused by the transmission 
tion performed by the management cell As in a regular links between the nodes in order to arrive at the total time 
OAM celt a 10-bit EDC field 8 contains an error detection that the cell has spent in the network. While this delay-stamp 
code that is computed over the information field of the cell. approach requires the new switch capability of local observ- 
In Che ATM ernbodiment of FIG. 1, the Function Type 50 abilityofpacketdelay.it has a big advwtage over the OAM 
field 6 contains a code that describes to the network the type ceil performance monitoring method in that it does not 
of performance monitoring function performed by the ATM require any time-of-day synchronization between the 
management celL labeling the cell as an "end-to-end delay switches in a network. 

accumulaung" cell, a "node-by-node delay measurement" For monitoring of packet delay in more detail, a node- 
cell an "end-to-end loss accumulation" cell a "node-by- 55 by-node delay measurement management packet may be 
node loss measurement" ceil, or as some combination of the used. A particular advantage of the nodeby-node delay 
previous types. Additional codes can be defined for the measurement management packet is that it allows the corn- 
monitoring of other performance parameters. The informa- putation and relay of instantaneous delay information. This 
tion field of the management cell will also optionally contain type of management packet contains multiple delay-stamp 
a 4-byte Timestamp field 9. a Delay-stamp field 10. and a eo fields for recording each local delay encountered on a 
CeU Loss Count field 12. as well as other function-specific node-bynode basis, as illustrated in FIG. 3 for an ATM 
fields 14 that are used for additional delay-stamps, cell loss embodiment. In FIG. 3. at each node the cell is stamped in 
counts etc.. as needed. These fields can also be used for the Tune stamp field 9. or in the separate internal routing 
monitoring and reporting other performance parameters. header, with the time of entry into the node. This time is then 
For measurement of packet delay, a switch is required that 65 compared to the time of exit from Aat node and the 
can me^e the differeWbetween the arrival time and the calculated delay is recorded Lid tc , Ac- ceil in one of a m^tmk 
Satire time of any packet at that switch, a capability of unused May-stamp fields 22. This allows monitoring of 
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the delay performance of each intermediate node at the management cell, the total cell loss is then calculated at the 
virtual connection endpoint. If desired the end-to-end cell endpoint of the virtual connection. A combined loss man- 
delay can also be computed at the destination endpoint agement packet can be used to allow each node to record 
being the sum of the delay-stamp fields. Alternatively, both accumulated and node-by-node packet loss simulta- 
additional timestamp fields can be used for each node, 5 neously. Other useful information can be recorded in the 
allowing the switch to merely record in one timestamp field node-by-node loss measurement management packet, such 
the time that the packet arrives, and then record the time the as the positions of individual discarded packets within a data 
packet leaves into another timestamp field. The calculation packet block. 

of node-by-node and end-to-end delay can then be made The methods described require in several of the embodi- 

when the packet has reached the endpoint of the virtual 10 ments that the management packets experience the same 

connection. A further alternative is for die cell to be a delay and packet loss rate as the user data packets in a 

combination delay cell, allowing the switch to record both particular virtual connection. The simplest way to iraple- 

the accumulated and the node-by-node delays in the same ment this is to have the management packets follow the same 

packet path as the user packets through the switch. In order to 

Measurement of packet loss requires a switch with local 15 accomplish this, the payloads of the management packets 

observabiUty of packet loss, allowing the switch to count the must be modified without removing the packets from the 

number of packets it discards per virtual connection. Packet data packet stream. Management packets processed in this 

discards occur for various reasons, such as switch buffer manner are "associated management packets". Shown in 

overflow due to excessive traffic, corrupted routing infor- FIG. 6 is a block diagram of an embodiment of a switch with 

mationin a cell header field or on the virtual connection, and 20 the capability of processing the payloads of associated 

when packets have failed to arrive at their destination within management packets without removing them from the data 

a pre-specined time and have therefore "expired." Each packet stream. This embodiment also illustrates one of 

switch must count the number of packets it discards within several possible implementations of local observability of 

each virtual connection. This capability is already available packet delay. 

in certain existing switches, such as the AT&T GlobeView- 25 In PIG. 6, a data stream enters the switch 31 at input ports 
2000 switch [See AT&T. GlobeView-2000 Broadband Sys~ 32. The data stream consists of user data packets 34 inter- 
im Network: Network Management, 255-179-079, Issue spersed with management packets 36. The user packets 34 
1.0, Release 13. 1995J and the Newbridge 36150 Main- and management packets 36 enter the switch input header 
Street switch I See Newbridge, 36150 MainStreet ATMnet processor 38. which reads the network routing instructions 
Access Switch Technical Practices, NNP 95 -1602-0 1-00- A, 30 for each packet and uses them to create an internal routing 
Release 3.1, Generic A1U13, 1995J. header that is then attached to that packet. If the packet is a 

The management packet used to measure packet loss can delay management packet, creation of the internal routing 

be the same packet as used to measure packet delay or a header will include stamping of the time of entry into the 

different packet Packet loss is measured according to one internal routing header by the time-of-day processor 40. 

einbodiment of the present invention through me insertion of 35 While it is only necessary to stamp delay management 

end-to-end loss accumulating management packets into a packets with the time of entry, if it is more expedient in 

virtual connection between blocks of user packets, as Olus- terms of processor design, time, etc., it is not necessary that 

trated in FIG. 4 for an ATM erarjodiment Each end-to cod management packets 36 be distinguished from user data 

loss accumulating management cell contains a Cell Loss packets 34 by the input header processor 38. Instead, all 

Count field 12 that is initialized to zero. When nodel 24 40 incoming data packets on the input ports 32 can have a time 

receives the end-to-end loss accumulating management cell of entry recorded in their internal routing header, 

with a Cell Loss Count field value NO. it increments the Cell Once the internal routing header has been created, all the 

Loss Count field 12 by the number of cells discarded by the data packets are routed through the switch fabric 42 and 

node 24 in that virtual connection since the previous loss directed towards their intended output ports 48. Before 

management cell arrived in that particular virtual connec- 45 entering the output ports 48. the packets are routed through 

tion. The cell is then received by node2 26. which in ore- the output header processor 44. which removes the internal 

ments the Cell Loss Count field 12 by the number of cells routing header from the packets. At this time, the processor 

discarded within that virtual connection by that node. A 44 identifies the delay management packets 36 and uses the 

similar procedure is followed at the next node. node3 28, and time-of-day stamp in the internal routing header to calculate 

so on. By the time the end-to-end loss accumulating man- 50 the delay incurred by each packet in the switch. The time- 

agement cell ultimately arrives at its destination virtual of-day stamp in the internal routing header is compared to 

connection endpoint the Ceil Loss Count field 12 accurately the current time-of-day. and the difference is written or 

reflects the total number of discarded cells in the preceding incremented by the Delay-stamp processor 44 into the 

block of user cells. In contrast to the use of existing OAM delay- stamp field of the management data packet 36. while 

performance management cells, it is not required that a node 55 the previous user data packet 34 in the data stream is being 

know how many cells to expect in the user cell block, since sent to the appropriate output port 48. In this way. the 

each node accurately counts how many cells it discards. management data packet retains its place in the data stream. 

For monitoring of packet loss in greater detail node-by- In an alternate embodiment the delay-stamp processor 46 

node loss measurement management packets containing can merely record the time of exit directly into a timestamp 

multiple fields to record the number of discarded packets 60 field in the delay management packet 36, allowing the delay 

counted at each node may be used. An illustration of this is calculations to be made later when the packet has reached 

shown in FIG. 5 for an ATM embodiment. At each node, the the virtual connection endpoint 

number of cells discarded within that virtual connection by A similar procedure is used in a preferred embodiment to 

that node since the departure of the previous loss manage- enter packet loss counts into loss management packets. It is 

ment cell in that virtual connection is stamped at the nodes 65 not necessary that loss management packets be stamped with 

24. 26, 28 into one of a multiple of unused Cell Loss Count a time of entry to the switch by the time-of-day processor 40. 

fields 30. Just as for the node-by-node delay measurement but this may be done if it is more expedient to do so. When 
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a loss management packet is routed through the output management packets 54 enter the switch input header pro- 
header processor 44 the processor identifies it as a loss cessor 58. which reads the network routing instructions ; for 
™ernem packet The number of packets discarded by each packet and uses them to create an intern* I routing 
.esw^h^ r^l^ 

Loss management packet in that virtual connection was 5 ? ^ ^ ^ ^ tQ 
processed by the switch is *eo Sntrol processor 62 for modification. Once the internal 
the packet loss count field of the packet ^ writing can be ^ nca ders have been created by the input header 
done either as an additional function of the Delay-stamp $$ me data/associated management packets 52 
processor 46. or by a separate processor opcraUng in a m ^ swUch fabdc ^ ^ fa^fr 
similar manner. The packet loss count is written into the loss 10 the output headcf processor 64 towards their intended output 
management data packet 36 while the previous user data ^ ^ mcans raat me noo . assoc iated management 
packet 34 in the data stream is being sent to the appropriate ^ ^ . ^ ^ ^ stream whfle was being 
output port 48. allowing the Loss management data packet to J^^, by & c ^ol processor 62. 
retain its place in the data stream. ft h t0 usc non -associated management packets 
In an alternate ernbodiment the time-of-day processor 40 tQ ^j^m performance monitoring functions such as 
writes the time a delay management packet 36 arrives at the loca] observability of packet delay and packet losses. This is 
switch directly into a timestamp field in the packet rather Q^iemented in one embodiment by having the control 
than into the internal routing header. In this case, the processor 62 replace the non-associated management packet 
time-of-day processor 40 will function like the delay-stamp 54 mat ^ rcm0V ed from me data so-eam for rxo<^$ing with 
processor 46 in that the time of entry will be recorded in the 20 a dummy packet" that holds the place of the 
payload of the delay management packet 36 while the n0 n-associated management packet 54 in the data stream- 
previous user data packet 34 is being sent to the switch ^ comtA processor $2 then separately records the neces- 
fabric 42. The delay-stamp processor 46 then reads the time sajy performance monitoring information in the payload of 
of entry from the timestamp field of the delay management thft packet 54. When it has finished, the control 
packet and uses it to calculate the delay by which the 23 procesSOT 62 replaces the dummy marker packet in the data 
processor 46 increments the deUy-stamp field in the packet. strcam with the appropriate nonassociated management 

A switch capable of modifying packets without removing packet 54. 
them from their place in the data stream has uses beyond the Alternatively, local observability of such performance 
performance monitoring function. For example, such a ^ parameters as delay and packet loss can be implemented 
switch could be utilized to help recovery in the ATM u$ing aD alternate version of the control processor. In this 
Application Layer by allowing identification of from exactly eriibodiment. all data packets are routed through the control 
where in the data stream a cell has been lost. This informa- processor rather than just non-associated management pack- 
tion in turn can be used to request a rctransmiltal of that cell ets Parameters such as delay and packet loss count infor- 
ooly. consuming fewer resources than if an entire data block ^ mation are recorded into the payloads of appropriate man- 
had to be retransmitted. Alternatively, the same information agement packets, while dummy writes are made to the 
can be used to minimize the damage caused to the data by payloads of user data packets, leaving the user data 
the absence of the cell, without retransmittal. via error unchanged. Although this embodiment potentially requires 
concealment processes. some unnecessary write operations, it maintains the order of 
If management packets must for some reason be extracted ^ all packets in the data stream, and may provide efficiencies 
from the data stream and then diverted to a special processor, i n implementation or in other areas, 
as shown in one ernbodiment in FIG. 7, they do not Use of management packets provides an improvement 
experience the same delay as user data packets and will not over current OAM performance monitoring methods 
normally retain their positions relative to the user data because intermediate nodes along a connection can add their 
packets in the same virtual connection. In FIG. 7, a data 43 own local performance data into the payload of the packets, 
stream consisting of user data packets and "associated Management packets with delay stamping can be used to 
management packets" 52, interspersed with "non-associated accurately measure and report both the end-to-end packet 
management packets** 54, enters the switch 56 at input ports delay and the packet delay at each node along a connection, 
50. without the need for time-of-day clock synchronization 
Associated management packets are those management 50 between nodes. Management packets can also accurately 
packets that are intended to be processed in a manner that measure and report the packet loss at each node along a 
allows them to retain their place in the data stream, while connection, rninimizing the confusion created by misin- 
non-associated management packets are management pack- serted packets. Non-associated management packets allow 
ets that are processed in such a way that they lose their collection and reporting of cumulative performance and 
position with respect to the other packets in the data stream. 55 usage information from each node along a connection. The 
Non-associated management packets are used to gather combination of the management packets and switches 
cumulative information from each switch, such as the aver- described allow accurate and detailed QoS measurements 
age and maximum packet delays measured in the switch along a network connection. 

since the previous management packet. The control proces- Modifications and substitutions by one of ordinary skill in 

sor 62 accumulates a set of such performance or usage w the art are considered to be within the scope of the present 

measurements and records this information into the arriving invention, which is not to be limited except by the claims 

non-associated management packets 54. The ability to write which follow, 

into the payload of a non-associated type of packet is what is claimed is: 

available in existing switches, such as the GTE Government 1. a method for performance monitoring in an electronic 

Systems SPANet switch. 65 communications network having at least one network 

Id the ernbodiment of FIG. 7, the user data packets/ connection, each of said network connections having a 

associated management packets 52 and some non-associated network connection originating endpoint a network connec- 
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lion destination endpoint, and at least one network connec- 
tion intermediate node, comprising, in combination, the 
steps of: 

creating at least one management data packet, each of said 
management data packets comprising a header and a 
payload data field each of said management data 
packets further comprising, in combination: 
routing header means, said routing header means being 

part of said management data packet header; 
packet function field, said packet function field being 
part of either said management data packet header or 
said management data packet payload data field; and 
at least one payload local performance monitoring 
field, said payload local performance monitoring 
field being part of said management data packet 
payload data field; 
inserting at least one of said management data packets 
into a stream of user data packets at one of said network 
connection originating endpoints; 
measuring at least one performance monitoring parameter 
at each of said network connection intermediate nodes; 
for each of said management data packets, recording, at 
each of said network connection intermediate nodes 
while maintaining the position of said management 
data packet in the packet data stream, in one of said 
payload local performance monitoring fields in said 
management data packet said performance monitoring 
parameter measured at said network connection inter- 
mediate node; and 

extracting the values in said payload local performance 
monitoring fields after said management data packet 
reaches said network connection destination endpoint 

2. The method of claim 1, further comprising the steps, in 
combination, of: 

including in said payload local performance monitoring 

fields at least one payload delay-stamp field, initialized 

to a predetermined value; 
detecting entry of each of said management data packets 

into any of said network connection mtermediate 

nodes; 

for each of said management data packets and said 
network connection intermediate nodes, recording the 
time of entry of said management data packet into said 
network connection intermediate node; 

detecting the imminent exit of said management data 
packet from said network connection intermediate 
node; 

comparing the time of said imminent exit of said man- 
agement data packet to said recorded time of entry; and 

incrementing, at said network connection intermediate 
node while maintaining the position of said manage- 
ment data packet in the packet datastream, the value in 
one of said payload delay-stamp fields by the difference 55 
between said time of entry and said time of exit 

3. The method of claim 2, further including the step of 
adding the values in all said incremented ones of said 
payload delay-stamp fields of said management data packet 
in order to get a total accumulated delay value, when said 
management data packet reaches said network connection 
destination endpoint 

4. The method of claim 1, wherein each of said manage- 
ment data packets further include a packet type field, iden- 
tifying to said network connection intermediate nodes and 
said network connection destination endpoint that said data 
packet is a management data packet 
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5. The method of claim 1. wherein said routing header 
means of each of said management data packets contains a 
field identifying to said network connection intermediate 
nodes and said network connection destination endpoint that 
said data packet is a management data packet. 

6. The method of claim 2. further including the steps, in 
combination, of: 

including in said payload local performance monitoring 
fields at least one payload timestamp field; and 

for each of said delay management packets, recording, at 
said network connection intermediate node while main- 
taining the position of said management data packet in 
the packet datastream. into one of said payload times- 
tamp fields, the time of entry of said management 
packet into said network connection intermediate node. 

7. The method of claim 6. further including the step of 
adding the values in all said incremented ones of said 
payload delay-stamp fields of said management packet, in 
order to get a total accumulated delay value, when said 
management packet reaches said network connection desti- 
nation endpoint 

8. The method of claim 6, further including the step of 
recording, at said network connection intermediate node 
while maintaining the position of said management data 
packet in the packet datastream. said time of irtiminent exit 
in an unused one of said payload timestamp fields of said 
delay management packet 

9. The method of claim 8. further comprising the step of 
comparing each of said payload timestamp fields containing 
said times of entry of said delay management packet into 
each of said network connection intermediate nodes to the 
corresponding one of said payload timestamp fields contain- 
ing said time of imminent exit from said network connection 
intermediate node in order to obtain a delay value for each 
of said network connection intermediate nodes. 

10. The method of claim 2. wherein said time of entry of 
said management packet into said network connection inter- 
mediate node is recorded in an internal routing header 
created at said network connection intermediate node and 
attached to said management packet while in said network 
connection intermediate node. 

11. The method of claim 10. further including the step of 
adding the values in all said incremented ones of said 
payload delay-stamp fields of said management packet, in 
order to get a total accumulated delay value, when said 
management packet reaches said network connection desti- 
nation endpoint 

12. The method of claim 1. further comprising the steps, 
in combination, of: 

including in said payload local performance monitoring 
fields at least one payload discarded packet count field, 
initialized to a predetermined value; 

counting at each of said network connection intermediate 
nodes the number of said user data packets that are 
discarded by said network connection intermediate 
node between the departure of an immediately previous 
one of said management data packets and the departure 
of a current one of said management data packets; and 

incrementing, at said network connection intermediate 
node while maintaining the position of said manage- 
ment data packet in the packet datastream. the value in 
one of said payload discarded packet count fields of 
said current one of said management data packets by 
said discarded number of said user data packets. 

13. The method of claim 12. further comprising the step 
of adding the values in all said incremented ones of said 
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payload discarded packet count fields of said management 
data packet in order to gel a total accumulated discarded 
user packet value, when said management data packet 
reaches said network connection destination endpoint 

14. The method of claim 2. further comprising the steps. 5 
in combination, of: 

including in said payload local performance monitoring 
fields at least one payload discarded packet count field, 
initialized to a predetermined value; ^ 

counting at each of said network connection intermediate 
nodes the number of said user data packets that are 
discarded by said network connection intermediate 
node between the departure of an immediately previous 
one of said management data packets and the departure ^ 
of a current one of said management data packets; and 

incrementing, at said network connection intermediate 
node while maintaining the position of said manage- 
ment data packet in the packet datastream. the value in 
one of said payload discarded packet count fields of M 
said current one of said management data packets by 
said discarded number of said user data packets. 

15. The method of claim 14, further comprising the step 
of adding the values in all said incremented ones of said 
payload discarded packet count fields of said management 25 
data packet, in order to get a total accumulated discarded 
user packet value, when said management data packet 
reaches said network connection destination endpoint 

16. The method of claim 15. further comprising the step 

of adding the values in all said incremented ones of said ^ 
payload delay-stamp fields of said management data packet, 
in order to get a total accumulated delay value, when said 
management data packet reaches said network connection 
destination endpoint 

17. A router for use in an electronic communications 35 
network comprising, in combination: 

at least one input for receiving data packets from at least 

one network connection; 
at least one output for relaying data packets to one of said 

network connections; 40 
means for detection of entry of a management packet into 

said router, said management packet being comprised 

of at least a header field and a payload field; 
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means for associatively recording the time of entry of said 
management packet into said router, wherein said 
means for associatively recording said lime of entry 
records said lime of entry in a payload timestamp field 
in the payload field of said management packet; 

means for detection of the imminent exit of said manage- 
ment packet from said router; and 

means for recording, at said router, a parameter into the 
payload field of said management packet that can be 
used to obtain the delay incurred by said management 
packet in said router. 

18. The router of claim 17. further including means for 
recording, at said router, said time of inuninent exit of said 
management packet from said router into the payload field 
of said management packet 

19. A performance-monitored switching mechanism for 
concurrently routing each of a plurality of data packets in a 
datastream. each of said data packets comprising a header 
and a payload, to a selected one of plural output data 
pathways in accordance with a destination address, said 
mechanism comprising, in combination: 

a plurality of input ports for concurrently receiving said 
data packets; 

a plurality of output data pathways; 

switching means connected between said input ports and 
said output data pathways; and 

means for modifying said payload of one of said data 
packets without said modified data packet losing its 
position in said datastream of said data packets, 
wherein said means for modifying said payload of one 
of said data packets further comprises, in combination: 
means for selecting one of said data packets in said 

datastream to be modified in said payload; 
means for writing data to said payload of said selected 

data packet; and 
means for performing dummy writes to non-selected 
ones of said data packets in said datastream. 

***** 
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